Two Modes are Better Than One: a Multimodal Assessment Framework Integrating Student Writing and Drawing

We are beginning to see the emergence of advanced automated assessment techniques that evaluate expressive student artifacts such as freeform written responses and sketches. These approaches have largely operated individually, each considering only a single mode. We hypothesize that there are synergies to be leveraged in multimodal assessments that can integrate multiple modalities of student responses to create a more complete and accurate picture of a student’s knowledge. In this paper, we introduce a novel multimodal assessment framework that integrates two techniques for automatically analyzing student artifacts: a deep learning-based model for assessing student writing, and a topology-based model for assessing student drawing. An evaluation of the framework with elementary students’ writing and drawing assessments demonstrate that 1) each of the framework’s two modalities provides an independent and complementary measure of student science learning, and 2) together, the multimodal framework significantly outperforms either uni-modal approach individually, demonstrating the potential synergistic benefits of multimodal assessment.